In preparation for sylabs/singularity#1021, where we will add a placeholder OCI runtime
launcher, refactor the existing code so that:

* The internal/pkg/runtime/launcher package contains common option
  handling, utility functions, and a Launcher interface.
* There is a launcher.native package containing the existing launch
  code for the native singularity runtime.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Accepts no options, fails if options provided.

Implements an Exec method which does nothing.

Fixes sylabs/singularity#1021

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Handle empty structs (not just nils).

Handle always-set network, config file.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Replace invocations of SingularityCE's OCI runtime engine with wrapped
invocations of runc.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

As noted in review, the `runc` call for `oci exec` was not actually
being made. E2E test for `oci exec` was only checking return code, not
for output so it passed. Fix these issues.

In `e2e/suite.go` also move registry creation so that it is only
performed for the man tests in PID+mount NS. It is not needed for the
OCI/CGROUPS e2e tests and wastes CI time.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

As suggested in review the streams can be straight `io.Writer` /
`io.Reader`.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

In preparation for further OCI runtime work, move the code that calls
out to runc/conmon from internal/app/singularity into
internal/pkg/runtime/launcher/oci

The oci.Launcher will make use of the basic OCI operations to run
containers from its Exec function, so this is a good location while
that work proceeds. The functions will be likely be modified
considerably, and potentially moved again in future, as the
design/implementation of the OCI runtime interaction is developed.

The internal/app/singularity OCI* functions are left as a minimal shim
layer, between the CLI layer and the launcher, at this time.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

As a first step toward run/shell/exec actions on native OCI images,
implement a minimal `singularity run --oci mybundle` which:

* Requires an on-disk bundle with appropriate `config.json`.
* Runs this bundle using `crun` or `runc`.
* Makes no attempt to handle any arguments or options.
* Does not modify the `config.json` - i.e. it must match namespace /
  mapping requirements for rootless execution etc.

At this stage, the functionality is essentially equivalent to
`singularity oci run` and is not yet useful.

The primary purpose of the PR is to refactor some of the code that
passes args for launching a container.

In addition, we now use `crun` in preference to `runc` if
available. `crun` supports e.g. single uid->uid mapping in a
usernamespace (without root mapping).

Closes sylabs/singularity#598

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Add subtest information to the large multi-step OCI test functions for
easier debugging.

Move the test of `oci help` out of the SEQ/ordered section. It can run
in parallel, and only once regardless of cgroups managers available.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

The Ubuntu packaged crun is very old. Use a release from GitHub instead.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When running containers in `--oci` mode, the argument to run is now an
image reference corresponding to a native OCI format handled by
containers/image, i.e.

* docker://
* docker-archive:
* docker-daemon:
* oci-archive:
* oci:

The source image is extracted into a temporary OCI bundle, with a
minimally valid configuration that:

* Runs the process specified by CMD & ENTRYPOINT only.
* Sets the environment specified by the image ENV only.

The approach is very naive - we pull through Singularity's OCI blob
cache into a temporary oci layout dir, before creating the bundle from
it. Auth handling for registries is not yet wired up. There is
duplication of various pieces of code from the build / SIF OCI flows
as these are not easily exposed to the area we are working in.

The intent of the PR, at this stage, is simply to allow e.g.

    singularity run --oci docker://sylabsio/lolcow

Closes sylabs/singularity#1036

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Simple test to create a native oci bundle from the supported sources,
and verify it is valid with runtime-tools/validate.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When using `run` or `exec` with the `--oci` runtime mode, accept
arguments on the command line.

For `run`, the arguments override any CMD specified by the image.

For `exec`, the arguments replace ENTRYPOINT/CMD entirely, bypassing
the process configuration in the image config.

This mirrors the behavior of Singularity images today, via the exec
and run runscripts - but is implemented in the OCI bundle config,
rather than a script in the container.

Closes sylabs/singularity#1024

Closes sylabs/singularity#1092

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Enable `singularity shell --oci ...` with behavior matching native
runtime, i.e.

* Run shell set with SINGULARITY_SHELL or --shell
* If not set, try /bin/bash --norc
* If not available, use /bin/sh

Closes sylabs/singularity#1025

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Ensure e2e tests for oci actions use user profile.

Set uid/gid mappings explicitly. We need to do this anyone, going
forward, but here it works around:

containers/crun#1072
Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

runc -> crun

Need uidmap on Debian

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Tidy up the rpm spec file.

* Remove the manual handling of build root, GOPATH etc. Not needed as
  we are using go modules now.
* Remove redundant explicit deps.
* Fix crun -> runc dep for EL7.
* Use rpm make_xxx macros instead of direct make calls.
* Ensure all directories created are owned by package.

Fixes sylabs/singularity#1142
Fixes sylabs/singularity#1141

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Co-authored-by: Mike Frisch <mikef17@gmail.com>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

…1530)

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

…st of this)

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

* oci: support --overlay

* support for multiple overlays, other revisions

* fixup: deduplicate RunWrapped logic

* lots of refactoring and cleanup

* remove leftover debug-related panic call

* cleanup comments, fix small issues w/erroring

---------

Co-authored-by: David Trudgian <david.trudgian@sylabs.io>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

(sylabs/singularity#1678)

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

When runc/crun is called from oci_runc_linux.go, Stdin was incorrectly
connected for the various runc/crun operations.

* Non-interactive operations such as resume / kill don't need Stdin.
* Interactive operations (run/exec) had cmd.Stdin incorrectly set to
  os.Stdout. This prevented OCI containers from receiving input from
  pipes, redirection, etc.

Fixes sylabs/singularity#1712

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

These were missed, resulting in sylabs/singularity#1712 not being caught by the e2e suite.

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

OCI-mode: support mounting of squashfs images (read-only) and extfs images (read-write or read-only) with --overlay, alongside the already existing functionality of directory-based overlays.

Squashed merge of:

* prepare data structures for non-dir overlays

* fix e2e test (no more "auto-create overlay dir" functionality)

* switch to using image.Init() to analyze image files

* added caching mechanism to FindBin

* first working version with squashfs support

* refactor funcs into methods of OverlayItem where appropriate

* standardized naming + added lots of comments

* addressing first round of review comments

* refactor: move non-OCI-dep. code to internal/pkg/util/fs/overlay

* initial support for extfs overlay (readonly)

* resurrect testing of write to persistent overlay

* fix bug in writable overlay, introduced in course of refactor

* removed caching mechanism from FindBin

* addressing second round of review comments, except tests

* move calling of prepareWritableOverlay() into Item.Mount()

* removed redundant error-string wrapping

* overlay.Item unit-test

* overlay.Set unit-test

* e2e tests for oci image overlays, misc. testing improvements

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

fix rel. path --workdir with --scratch, add oci e2e tests, from sylabs 1694

…d overlays (sylabs/singularity#1730)

* fallback to fuse-overlayfs if kernel doesn't support unpriv overlays

* addressed first round of review comments

* fix code-flow in UnprivOverlaysSupported() for unsupported case

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

(sylabs/singularity#1740)

* oci: support for writable extfs img overlay via fuse-overlayfs

* added unit- and e2e-tests

* fixed exposure of "upper" & "work" subdir in readonly overlays

* added fix for file-ownership in FUSE-mounted images + e2e test of fix

* fix handling of "upper" in :ro overlay dirs, adjust e2e tests

Signed-off-by: Edita Kizinevic <edita.kizinevic@cern.ch>

Signed-off-by: Dave Dykstra <2129743+DrDaveD@users.noreply.github.com>